Information processing apparatus, information processing method, and storage medium

ABSTRACT

Whether a reference map can be changed is determined based on an index for evaluating a user&#39;s unlikeliness to notice a change in an appearance of a virtual object in an image captured by a camera, the change occurring when the reference map is changed to another map.

BACKGROUND

Field

The present disclosure relates to an information processing apparatus,an information processing method, and a program, and more particularlyto ones suitably used to superimpose and display a virtual object on animage captured by an imaging apparatus.

Description of the Related Art

There is a mixed reality (MR) technology. The MR technology is atechnology that uses a camera and a device implementing a display, suchas a head-mounted display (HMD) and a tablet personal computer (PC), tosuperimpose a virtual object on an image captured by the camera anddisplay the resultant on the display.

In the MR technology, a superimposing position of the virtual object inthe image is usually determined by using a position and orientation ofthe camera, which is estimated by associating the image captured by thecamera with a map representing a spatial arrangement of an index beingassociated with an index in the image in the scene. In general, the mapand the processing for associating the map with the image captured bythe camera include errors. The position and orientation of the cameraestimated by associating the map with the image captured by the cameratherefore contains errors. Such errors can change the superimposingposition of the virtual object and give a sense of incongruity to theuser. Japanese Patent No. 3793158 discusses a technique for correctingthe superimposing position of the virtual object in consideration of atime-series history of the position and orientation of the camera.

Errors in the processing for associating the map with the image capturedby the camera cause changes in a display position of the virtual objectwith relatively high frequency. Such changes can be suppressed by thetechnique discussed in Japanese Patent No. 3793158. On the other hand, achange in the display position of the virtual object due to a change ofthe map to be referred to during association occurs sporadically. Thetechnique discussed in Japanese Patent No. 3793158 is therefore not ableto remove such a change in the display position of the virtual object,and a sense of incongruity may be given to the user when the map ischanged.

SUMMARY

The present disclosure is directed to reducing a sense of incongruitygiven to the user when the map referred to to determine thesuperimposing position of the virtual object on the captured image ischanged.

According to an exemplary embodiment, an information processingapparatus for performing processing for superimposing a virtual objecton an image captured by an imaging apparatus based on a result ofassociation between a map and the image, the map representing anarrangement of an index being associated with an index included in animage which is captured, includes a determination unit configured todetermine whether the map is able to be changed, based on an appearanceof the virtual object superimposed on the image captured by the imagingapparatus; and a changing unit configured to change the map in casewhere the determination unit determines that the map is able to bechanged.

According to another exemplary embodiment, an information processingapparatus for performing processing for superimposing a virtual objecton an image captured by an imaging apparatus based on a result ofassociation between a map and the image, the map representing anarrangement of an index being associated with an index in an image whichis captured, includes a determination unit configured to determinewhether the map is able to be changed, based only on a temporal changein a position and orientation of the imaging apparatus; and a changingunit configured to change the map in case where the determination unitdetermines that the map is able to be changed.

Further features will become apparent from the following description ofexemplary embodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of hardware of aninformation processing apparatus.

FIG. 2 is a diagram illustrating a first example of a configuration of amixed reality (MR) system.

FIG. 3 is a flowchart illustrating a first example of processing of theMR system.

FIG. 4 is a diagram illustrating a second example of the configurationof the MR system.

FIG. 5 is a diagram illustrating a third example of the configuration ofthe MR system.

FIG. 6 is a flowchart illustrating a second example of the processing ofthe MR system.

FIG. 7 is a diagram illustrating a fourth example of the configurationof the MR system.

FIG. 8 is a flowchart illustrating a third example of the processing ofthe MR system.

DESCRIPTION OF THE EMBODIMENTS

Before describing exemplary embodiments, an example of a configurationof hardware that implements an information processing apparatusdescribed in each exemplary embodiment will be described with referenceto FIG. 1.

FIG. 1 is a diagram illustrating an example of a configuration ofhardware of an information processing apparatus 100. In FIG. 1, acentral processing unit (CPU) 110 controls devices connected to a bus160 in a centralized manner. The CPU 110 reads and executes processingsteps and programs stored in a read-only memory (ROM) 120. For example,various processing programs and device drivers according to the presentexemplary embodiment, including an operating system (OS), are stored inthe ROM 120. The CPU 110 temporarily stores the processing programs anddevice drivers stored in the ROM 120 into a random access memory (RAM)130, and executes the processing programs and device drivers. An inputinterface (I/F) 140 inputs an input signal from an external apparatus(such as a display and an operation device) in a format processable bythe information processing apparatus 100. An output I/F 150 outputs anoutput signal to an external apparatus (display) in a format processableby the apparatus.

Next, a first exemplary embodiment will be described.

In the present exemplary embodiment, an MR system that uses simultaneouslocalization and mapping (SLAM) for simultaneously performing generationof a map and calculation of a position and orientation of a camera willbe described as an example. In SLAM, the generation and update of a mapare sporadically performed. In the present exemplary embodiment, whengenerating a new map, the information processing apparatus 100determines whether a change in a display position of a virtual object,occurring from switching of maps is unnoticeable to a user. If thechange is determined to be unnoticeable to the user, the informationprocessing apparatus 100 immediately switches the map to be referred toto the generated new map. On the other hand, if the change is determinedto be noticeable to the user, the information processing apparatus 100stores the new map without switching the maps. After a lapse of acertain time, the information processing apparatus 100 determines againwhether the change is unnoticeable to the user. Such switching of mapsin timing when the user is unlikely to notice a change in the displayposition of the virtual object can reduce a sense of incongruity givento the user.

In the present exemplary embodiment, to determine whether a change inthe display position of the virtual object is unnoticeable to the user,the information processing apparatus 100 obtains field of viewinformation which represents an appearance of the virtual object as seenfrom the point of view of a camera. Based on the field of viewinformation, the information processing apparatus 100 then determineswhether the virtual object is included in the field of view of thecamera. If the virtual object is not included, the informationprocessing apparatus 100 determines that the map to be referred to canbe changed. Details of the present exemplary embodiment are describedbelow.

FIG. 2 is a diagram illustrating an example of a module configuration ofthe information processing apparatus 100 according to the presentexemplary embodiment and the MR system in which the informationprocessing apparatus 100 is implemented.

An HMD 200 includes a camera 201 and a display 202. The user wears theHMD 200 and moves freely in an MR experience space. In the meantime, animage input unit 210 sequentially receives frame images continuouslycaptured by the camera 210. A camera position and orientation managementunit 211 associates a reference map 212, which is the map currentlyreferred to, with a frame image received by the image input unit 210 tocalculate a position and orientation of the camera 201 at the imagingtime of the frame image. In the present exemplary embodiment, a mapincludes three-dimensional points representing surfaces of objects lyingaround and luminance information about pixels in a predetermined rangearound the three-dimensional points in the image in which thethree-dimensional points are observed.

A field of view information management unit 214 manages and sequentiallypasses the field of view information to an image drawing unit 217 andthe information processing apparatus 100 (field of view acquisition unit221). The field of view information includes at least either one ofgeometric information and optical information for determining theappearance of the virtual object to the camera 201. For example, thefield of view information includes the position and orientation of thecamera 201 obtained from the camera position and orientation managementunit 211, virtual object information 215, and internal parameters 216 ofthe camera 201. The virtual object information 215 includes a positionand orientation of the virtual object, shape information (such as vertexinformation which is information indicating vertexes of polygonsexpressing a shape, and surface information which is informationindicating surfaces of the polygons), and optical information (such astransparency of the virtual object).

The image drawing unit 217 generates an image by superimposing thevirtual object on the frame image received from the image input unit 210based on the field of view information received from the field of viewinformation management unit 214, and outputs the generated image to thedisplay 202. The foregoing series of processes up to the drawing of theimage to the display 202 is performed in real time on each of the frameimages continuously transmitted from the camera 201. Meanwhile, a mapmanagement unit 219 generates a map by using a plurality of stored frameimages obtained from the image input unit 210, and the position andorientation of the camera 201 corresponding to each frame image,obtained from the camera position and orientation management unit 211.

The map management unit 219 stores the generated map as a reference mapswitching candidate 213. When generating a map, the map management unit219 initially calculates, by the principle of triangulation, thepositions of the three-dimensional points of characteristic pixels thatcan be associated between a plurality of frame images. The mapmanagement unit 219 then attaches luminance information about apredetermined range around the three-dimensional points in the frameimage having the latest imaging time among the plurality of frame imagesin which the three-dimensional points are observed, to the positions ofthe three-dimensional points and stores the resultant as a matrix. Themap management unit 219 checks a reference map change determinationresult 218 which is output by the information processing apparatus 100.If the reference map change determination result 218 indicates that themap can be switched, the map management unit 219 updates the referencemap 212 with the reference map switching candidate 213.

The information processing apparatus 100 according to the presentexemplary embodiment includes the field of view information acquisitionunit 221, a reference map change determination unit 222, and adetermination result output unit 223. The field of view informationacquisition unit 221 obtains the position and orientation of the camera201, the virtual object information 215, and the internal parameters 216of the camera 201 as the field of view information from the field ofinformation management unit 214. Based on the field of view information,the reference map change determination unit 222 determines whether themap currently referred to (reference map 212) can be changed. Thedetermination result output unit 223 outputs the latest result ofdetermination made by the reference map change determination unit 222 asthe reference map change determination result 218.

FIG. 2 illustrates an example where the image input unit 210, the cameraposition and orientation management unit 211, the field of viewinformation management unit 214, and the image drawing unit 217 areincluded in the information processing apparatus 100. However, at leastsome of these units may be included in an apparatus other than theinformation processing apparatus 100 (for example, the HMD 200).

Next, an example of processing of the MR system will be described indetail with reference to the flowchart of FIG. 3.

(Step S301)

In step S301, initialization processing of the MR system is performed.Specifically, the user previously arranges a marker board which can beautomatically detected from an image and on which feature points arearranged with known dimensions, in a scene in which the MR system isused. The user makes an operation on the camera 201 of the HMD 200 tocapture an image of the marker board. The map management unit 219obtains the image captured by the camera 201 based on the operation. Themap management unit 219 calculates the three-dimensional positions ofthe feature points on the marker board from the obtained image as thepositions of the three-dimensional points, and sets the positions of thethree-dimensional points and luminance information about surroundingpixels as an initial reference map 212. The information processingapparatus 100 activates loop processing of and after step S311 forexecuting map calculation processing, and loop processing of and afterstep S321 for executing map switching processing. The loop processing ofand after step S311 and the loop processing of and after step S321 areexecuted asynchronously with the processing of steps S301 to S309. TheHMD 200 obtains the virtual object information 215 and the internalparameters 216 of the camera 201 from a previously-stored memory area.

(Step S302)

Step S302 is loop processing to be repeatedly performed thereafter untilthe user ends the MR experience. The loop processing is executed at aframe rate such that the user can discern the processing of steps S303to S308 in real time.

(Step S303)

In step S303, the image input unit 210 obtains an image from the camera201 of the HMD 200.

(Step S304)

In step S304, the camera position and orientation management unit 211associates the image obtained in step S303 with the reference map 212 tocalculate the position and orientation of the camera 201 when the imageis captured. A specific example will be described. The camera positionand orientation management unit 211 multiplies the three-dimensionalpoints and the luminance information nearby, stored in the reference map212, by a position and orientation of the camera 201 and an internalparameter matrix of the camera 201. The camera position and orientationmanagement unit 211 thereby superimposes the luminance information(luminance values) on a region corresponding to a region in which theluminance information exists, among regions of the image obtained instep S303. The camera position and orientation management unit 211calculates, by optimization calculation, the position and orientation ofthe camera 201 that minimizes the sum of differences between theluminance values of the pixels of the two regions.

(Step S305)

In step S305, the image drawing unit 217 generates an image bysuperimposing the virtual object on the image obtained in step S303, andoutputs the generated image to the display 202 of the HMD 200. Aspecific example will be described. The image drawing unit 217 initiallyreceives the position and orientation of the camera 201 calculated instep S304 and the virtual object information 215 from the field of viewinformation management unit 214. The image drawing unit 217 calculates arelative position and orientation of the virtual object with respect tothe camera 201 based on the position and orientation of the camera 201and the position and orientation of the virtual object. The imagedrawing unit 217 then multiplies the shape information about the virtualobject by the relative position and orientation of the virtual objectand the internal parameter matrix of the camera 201 to determine thedisplay position of the virtual object in the image captured by thecamera 201. The image drawing unit 217 then performs rendering in thedetermined position according to the optical information about thevirtual object.

(Step S306)

In step S306, the field of view information acquisition unit 221 obtainsthe following pieces of information from the field of view informationmanagement unit 214 as the field of view information for determining theappearance of the virtual object as seen from the point of view of thecamera 201. That is, the field of view information acquisition unit 221obtains the position and orientation of the camera 201, the virtualobject information 215, and the internal parameters 216 of the camera201.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306. Here, the field of view informationobtained in step S306 includes the position and orientation of thecamera 201, the position and orientation of the virtual object, theshape information about the virtual object (vertex information aboutpolygons), and the internal parameters 216 of the camera 201. In thepresent exemplary embodiment, if the virtual object is not included inthe field of view (imaging range) of the camera 201 from the point ofview of the camera 201 when the image is captured, the reference mapchange determination unit 222 determines that the reference map 212 canbe changed. If not, the reference map change determination unit 222determines that the reference map 212 is not able to be changed.

A specific example will be described. The reference map changedetermination unit 222 initially calculates the relative position andorientation of the virtual object with respect to the camera 201 basedon the position and orientation of the camera 201 and the position andorientation of the virtual object. The reference map changedetermination unit 222 multiplies three-dimensional coordinates of eachvertex of the polygons expressing the shape of the virtual object by therelative position and orientation of the virtual object and the internalparameters 216 of the camera 201. The reference map change determinationunit 222 thereby calculates a two-dimensional position of each vertex ofthe virtual object on a camera image projection plane (plane ofprojection of the image captured by the camera 201). If none of thetwo-dimensional positions of the vertexes of the virtual object on thecameral image projection plane falls within the field of view of thecamera 201, the reference map change determination unit 222 determinesthat the reference map 212 can be changed. If any, the reference mapchange determination unit 222 determines that the reference map 212 isnot able to be changed. The reference map change determination unit 222stores the determination result.

(Step S308)

In step S308, the determination result output unit 223 outputs thedetermination result of step S307 to a memory area where the referencemap change determination result 218 is stored.

(Step S309)

In step S309, end processing of the MR system is performed.Specifically, the information processing apparatus 100 ends the loopprocessing of and after step S311 and the loop processing of and afterstep S321. The information processing apparatus 100 frees the resourcesreserved in executing the programs for performing the flowchartillustrated in FIG. 3.

Next, an example of the loop processing of and after step S311,representing the map calculation processing activated in step S301, willbe described.

(Step S311)

Step S311 is loop processing to be repeatedly performed after processingis activated in step S301, until the end processing of step S309 isexecuted.

(Step S312)

In step S312, the map management unit 219 calculates a map withreference to pairs of a predetermined number of images obtained andstored in step S303 and the positions and orientations of the camera 201corresponding to the respective images, calculated in step S304. The mapmanagement unit 219 outputs the calculated map to a memory area storingthe reference map switching candidate 213.

Next, an example of the loop processing of and after step S321,representing the map switching processing activated in step S301, willbe described.

(Step S321)

Step S321 is loop processing to be repeatedly performed after processingis activated in step S301, until the end processing of step S309 isexecuted.

(Step S322)

In step S322, the map management unit 219 checks the memory area storingthe reference map switching candidate 213 to determine whether there isa reference map switching candidate 213. If there is a reference mapswitching candidate 213 (YES in step S322), the processing proceeds tostep S323. If there is no reference map switching candidate 213 (NO instep S322), the processing returns to the beginning (step S321) of theloop processing. The map management unit 219 performs the loopprocessing again after a wait of certain time.

(Step S323)

In step S323, the map management unit 219 checks the memory area storingthe reference map change determination result 218 to determine whetherthe reference map 212 can be changed. If the reference map 212 isdetermined to be able to be changed (YES in step S323), the processingproceeds to step S324. On the other hand, if the reference map 212 isnot able to be changed (NO in step S323), the processing returns to thebeginning of the loop processing (step S321). The map management unit219 performs the loop processing again after a wait of certain time.

(Step S324)

In step S324, the map management unit 219 switches the reference map212. A specific example will be described. The map management unit 219sets the map stored as the reference map switching candidate 213 to bethe reference map 212, and then clears the memory area storing thereference map switching candidate 213.

As described above, in the present exemplary embodiment, whether thereference map 212 can be changed is determined based on whether thevirtual object is included in the field of view of the camera 201. Thiscan reduce the sense of incongruity given to the user due to a changeoccurring in the display position of the virtual object when thereference map 212 is switched during MR experience.

[Modification 1-1]

The present exemplary embodiment has described an example where, in stepS307, the reference map 212 is determined to be changed if the virtualobject is not included in the field of view of the camera 201. However,this is not necessarily restrictive. Whether to change the reference map212 may be determined based on the ratio of (the area of) the displayregion of the virtual object to (the area of) the image captured by thecamera 201 on which the virtual object is superimposed. The smaller theratio of the display region of the virtual object to the image capturedby the camera 201 is, the less likely the user is to notice a changeoccurring in the position of the virtual object when the reference map212 is changed. Changing the reference map 212 when the ratio of thedisplay region of the virtual object to the image captured by the camera201 is small therefore provides the following effect. That is, even ifthe virtual object lies in the field of view of the camera 201 for along period of time, the reference map 212 can be changed in timing whena sense of incongruity given to the user is relatively small.

Next, an example of the processing of step S307 different from that ofthe foregoing first exemplary embodiment will be described as processingof the information processing apparatus 100 according to the presentmodification.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306. Here, the position and orientation ofthe camera 201, the position and orientation of the virtual object, theshape information about the virtual object (vertex information andsurface information about polygons), and the internal parameters 216 ofthe camera 201 are used as the field of view information obtained instep S306. In the present modification, if the ratio of the displayregion of the virtual object to the image captured by the camera 201 issmaller than a predetermined value, the reference map changedetermination unit 222 determines that the reference map 212 can bechanged. If not, the reference map change determination unit 222determines that the reference map 212 is not able to be changed. Thereference map change determination unit 222 stores the determinationresult.

A specific example will be described. The reference map changedetermination unit 222 initially calculates the relative position andorientation of the virtual object with respect to the camera 201 basedon the position and orientation of the camera 201 and the position andorientation of the virtual object. The reference map changedetermination unit 222 multiplies the surface information about eachsurface of the polygons expressing the shape of the virtual object bythe relative position and orientation of the virtual object and theinternal parameter matrix of the camera 201 to project the virtualobject onto the camera image projection plane. The reference map changedetermination unit 222 then counts the number of pixels where thesurfaces of the polygons of the virtual object exist in the region ofthe field of view range of the camera 201 on the camera image projectionplane. The counted total number of pixels is assumed to be the area ofthe display region of the virtual object. The reference map changedetermination unit 222 calculates the ratio of the obtained area to allthe pixels of the image captured by the camera 201. If the calculatedratio is smaller than or equal to a predetermined value, the referencemap change determination unit 222 determines that the reference map 212can be changed. If not, the reference map change determination unit 22determines that reference map 212 is not able to be changed.

As described above, according to the present modification, the referencemap 212 is determined to be able to be changed if the ratio of thedisplay region of the virtual object to the image captured by the camera201 is small. Consequently, even if the virtual object remainsconstantly in the field of view of the camera 201, the reference map 212can be switched in timing when a sense of incongruity given to the useris relatively small.

[Modification 1-2]

In the present exemplary embodiment and modification 1-1, whether thereference map 212 can be changed is described to be determined based onthe geometric appearance of the virtual object in the point of view ofthe camera 201 as the appearance of the virtual object. However, theappearance of the virtual object is not limited thereto. Whether thereference map 212 can be changed may be determined based on anappearance taking account of not only the geometric appearance of thevirtual object but the optical appearance of the virtual object as well.The reference map 212 can thus be changed by identifying a situationwhere the user is unlikely to notice a change in the display position ofthe virtual object (for example, where the virtual object superimposedon the image captured by the camera 201 blends well into the backgroundimage). In the present modification, a method for determining blendingof the virtual object into the background image based on the magnitudeof a luminance gradient of the image will be described as an example.

An example of the processing of step S307 different from that of theforegoing first exemplary embodiment will be described below asprocessing of the information processing apparatus 100 according to thepresent modification.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306 and the image captured by the camera201. Here, the position and orientation of the camera 201, the positionand orientation of the virtual object, the shape information (vertexinformation and surface information about polygons), the opticalinformation about the virtual object, and the internal parameters 216 ofthe camera 201 are used as the field of view information obtained instep S306. In the present modification, the reference map changedetermination unit 222 generates an image by superimposing the virtualobject on the image captured by the camera 201 based on the field ofview information, and calculates the magnitude of variations inluminance between the display region of the virtual object and itsvicinity. If the variations in luminance are smaller than apredetermined value, the reference map change determination unit 222determines that the reference map 212 can be changed. If not, thereference map change determination unit 222 determines that thereference map 212 is not able to be changed.

A specific example will be described. The reference map changedetermination unit 222 initially generates a partial image by extractingpixels within a predetermined range around the display region of thevirtual object in the image obtained by superimposing the object on theimage captured by the camera 201. The reference map change determinationunit 222 calculates a luminance gradient value of each pixel of thegenerated partial image. The reference map change determination unit 222calculates the ratio of the number of pixels having a luminance gradientvalue greater than a predetermined value to the number of pixels of thepartial image. If the calculated ratio is smaller than a predeterminedvalue, the reference map change determination unit 222 determines thatthe reference map 212 can be changed. If not, the reference map changedetermination unit 222 determines that the reference map 212 is not ableto be changed. The processing for generating the image by superimposingthe virtual object on the image captured by the camera 201 here can beimplemented by the same processing as that in which, in step S305, theimage drawing unit 217 generates the image by superimposing the virtualobject on the image obtained in step S303. Since the generated image isthe same as that generated by the image drawing unit 217, the imagegenerated by the image drawing unit 217 can be shared to save the amountof calculation, without the reference map change determination unit 222generating a new image.

By determining that the reference map 212 can be changed only if thereference map 212 is determined to be able to be changed by both themethod of the present exemplary embodiment or modification 1-1 and themethod of modification 1-2, not only the geometric appearance but theoptical appearance of the virtual object can also be taken intoconsideration. By doing so, the reference map 212 can be switched intiming when the virtual object blends well into the background as thetiming when the user is unlikely to notice a change in the displayposition of the virtual object. Whether the reference map 212 can bechanged may be determined only by the method of modification 1-2 withoutmaking a determination by the method of the present exemplary embodimentor modification 1-1.

Next, a second exemplary embodiment will be described. The firstexemplary embodiment has described an example in which whether thereference map 212 can be changed is determined based on the static fieldof view information obtained in a certain point of time. However,whether the reference map 212 can be changed may be determined based ona change in the field of view information obtained in a time series. Inthe present exemplary embodiment, a moving speed of the display positionof the virtual object is calculated as the change in the field of viewinformation. If the magnitude of the calculated moving speed is greaterthan a predetermined value, the reference map 212 is determined to beable to be changed.

The user is more likely to notice a change occurring in the displayposition of the virtual object due to switching of the reference map 212if the display position of the virtual object is stationary in the fieldof view. On the other hand, the user is less likely to notice the changeif the display position of the virtual object is moving. Therefore, bychanging the reference map 212 when the magnitude of the moving speed ofthe display position of the virtual object is large, the reference map212 can be changed in timing when the user is unlikely to noticeregardless of whether the virtual object lies in the field of view ofthe camera 201.

The present exemplary embodiment thus differs from the first exemplaryembodiment mainly in the method for determining whether the referencemap 212 can be changed. In the description of the present exemplaryembodiment, parts similar to those of the first exemplary embodimentwill be designated by the same reference numerals as in FIGS. 1 to 3. Adetailed description thereof will be omitted. Specifically, the presentexemplary embodiment and the first exemplary embodiment differ in theprocessing of step S307 in FIG. 3. An example of the processing of stepS307 different from that of the first exemplary embodiment will bedescribed below as processing of the information processing apparatus100 according to the present exemplary embodiment.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306. Here, the position and orientation ofthe camera 201, the position and orientation of the virtual object, theshape information about the virtual object (vertex information aboutpolygons), and the internal parameters 216 of the camera 201 are used asthe field of view information obtained in step S306. In the presentexemplary embodiment, the reference map change determination unit 222calculates the moving speed of the display position of the virtualobject as a change in the field of view information based on the fieldof view information obtained in a time series. If the magnitude of thecalculated moving speed of the display position of the virtual object isgreater than a predetermined value, the reference map changedetermination unit 222 determines that the reference map 212 can bechanged.

A specific example will be described. The reference map changedetermination unit 222 initially calculates a reference position of thevirtual object. The reference map change determination unit 222calculates maximum values and minimum values of the vertexes of thepolygons expressing the shape of the virtual object on X-, Y-, andZ-axes, and divides the sums of the maximum and minimum values on therespective axes by 2 to calculate three-dimensional coordinates as thereference position of the virtual object. Next, the reference map changedetermination unit 222 calculates the relative position and orientationof the virtual object with respect to the camera 201 based on theposition and orientation of the camera 201 and the position andorientation of the virtual object. The reference map changedetermination unit 222 then multiplies the reference position of thevirtual object by the relative position and orientation of the virtualobject and the internal parameter matrix of the camera 201 to projectthe virtual object onto the camera image projection plane. Thetwo-dimensional position on the camera image projection plane is therebyobtained. The reference map change determination unit 222 assumes theobtained two-dimensional position to be the display position of thevirtual object. The reference map change determination unit 222calculates the display position of the virtual object a plurality oftimes in a time series to calculate the moving speed of the displayposition of the virtual object in the image captured by the camera 201.If the magnitude of the calculated moving speed is greater than apredetermined value, the reference map change determination unit 222determines that the reference map 212 can be changed. If not, thereference map change determination unit 222 determines that thereference map 212 is not able to be changed.

As described above, in the present exemplary embodiment, the referencemap 212 is determined to be able to be changed if the magnitude of themoving speed of the display position of the virtual object is large.Even if the virtual object lies in the field of view of the camera 201for a long period of time, the reference map 212 can thus be switched intiming when a sense of incongruity given to the user is relativelysmall.

[Modification 2-1]

The present exemplary embodiment has described an example in whichwhether the reference map 212 can be changed is determined based on themoving speed of the display position of the virtual object on the cameraimage projection plane as a temporal change in the field of viewinformation. However, the temporal change in the field of viewinformation is not limited thereto. Whether the reference map 212 can bechanged may be determined based on a change in the orientation of thevirtual object as seen from the point of view of the camera 201. In thepresent modification, a change speed of the orientation of the virtualobject is calculated as a change in the field of view information. Thereference map 212 is determined to be able to be changed if themagnitude of the change speed of the orientation of the virtual objectis greater than a predetermined value. The user is more likely to noticea change in the display position of the virtual object if theorientation of the virtual object is stationary in the field of view. Onthe other hand, the user is less likely to notice a change in thedisplay position of the virtual object if the orientation of the virtualobject is changing in the field of view. Therefore, by changing thereference map 212 when the magnitude of the change speed of theorientation of the virtual object is large, the reference map 212 can bechanged in timing when the user is unlikely to notice even if thedisplay position of the virtual object is not moving.

An example of the processing of step S307 different from that of theforegoing second exemplary embodiment will be described below asprocessing of the information processing apparatus 100 according to thepresent modification.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306. Here, the position and orientation ofthe camera 201, the position and orientation of the virtual object, theshape information about the virtual object (vertex information aboutpolygons), and the internal parameters 216 of the camera 201 are used asthe field of view information obtained in step S306. In the presentmodification, the reference map change determination unit 222 calculatesthe change speed of the orientation of the virtual object as a change inthe field of view information based on the field of view informationobtained in a time series. If the magnitude of the calculated changespeed of the orientation of the virtual object is greater than apredetermined value, the reference map change determination unit 222determines that the reference map 212 can be changed.

A specific example will be described. The reference map changedetermination unit 222 multiples a rotation matrix representing theorientation out of the position and orientation of the virtual object byan inverse matrix of a rotation matrix representing the orientation outof the position and orientation of the camera 201. The reference mapchange determination unit 222 thereby obtains a rotation matrixrepresenting a relative orientation of the virtual object with respectto the camera 201. The reference map change determination unit 222calculates the rotation matrix representing the relative orientation tothe point of view of the camera 201 a plurality of times in a timeseries to calculate the change speed of the orientation of the virtualobject to the point of view of the camera 201 in the form of a rotationmatrix. The reference map change determination unit 222 then transformsthe rotation matrix into a rotation vector by Rodrigues's formula, andassumes the magnitude of the rotation vector to be the magnitude of thechange speed of the orientation of the virtual object. If the magnitudeof the calculated change speed of the orientation of the virtual objectis greater than a predetermined value, the reference map changedetermination unit 222 determines that the reference map 212 can bechanged. If not, the reference map change determination unit 222determines that the reference map 212 is not able to be changed.

As described above, according to the present modification, the referencemap 212 is determined to be able to be changed if the magnitude of thechange speed of the orientation of the virtual object is large. Even ifthe display position of the virtual object is not moving, the referencemap 212 can thus be switched in timing when the user is unlikely tonotice.

Next, a third exemplary embodiment will be described. The first andsecond exemplary embodiments have described examples in which whetherthe reference map 212 can be changed is determined based only on thefield of view information that is calculated by using the reference map212 currently referred to. However, whether the reference map 212 can bechanged may be determined by also using information about the referencemap to be switched to (reference map switching candidate 213) tocalculate the magnitude of a change in the appearance of the virtualobject from the point of view of the camera 201 when the reference map212 is actually switched. By actually calculating a change occurring inthe appearance of the virtual object when the reference map 212 isswitched, using the information about the reference map switchingcandidate 213 as well, the timing when a sense of incongruity given tothe user is small can be detected without omission.

The present exemplary embodiment thus differs from the first and secondexemplary embodiments mainly in the method for determining whether thereference map 212 can be changed. In the description of the presentexemplary embodiment, parts similar to those of the first and secondexemplary embodiments will be designated by the same reference numeralsas in FIGS. 1 to 3. A detailed description thereof will be omitted.

In the present exemplary embodiment, a change occurring in theappearance of the virtual object when the reference map 212 is switchedis calculated in the following manner. A display position of the virtualobject in the image captured by the camera 201 is calculated based onthe position and orientation of the camera 201 that are calculated byusing the reference map 212 currently referred to. A display position ofthe virtual object in the image captured by the camera 201 is alsocalculated based on the position and orientation of the camera 201 thatare calculated by using the reference map switching candidate 213. Adifference between the display positions of the virtual object is thencalculated as a change occurring in the appearance of the virtual objectwhen the reference map 212 is switched. If the magnitude of thecalculated difference in the display position of the virtual object issmaller than a predetermined value, the reference map 212 is determinedto be able to be changed. If not, the reference map 212 is determined tobe unable to be changed. Details of the present exemplary embodiment aredescribed below.

FIG. 4 is a diagram illustrating an example of a module configuration ofan information processing apparatus 400 according to the presentexemplary embodiment and an MR system in which the informationprocessing apparatus 400 is implemented. The functions of the componentsillustrated in FIG. 4 are not changes from those of the componentsillustrated in FIG. 2. Differences in the information exchanged betweenthe components will be described here.

Like the first exemplary embodiment, the camera position and orientationmanagement unit 211 stores a position and orientation of the camera 201that are calculated by associating the frame image received by the imageinput unit 210 with the reference map 212. In addition, the cameraposition and orientation management unit 211 stores a position andorientation of the camera 201 that are calculated by associating theframe image with the reference map switching candidate 213. The field ofview information management unit 214 has a function of aggregating andsequentially passing the two positions and orientations of the camera201, the virtual object information 215, and the internal parameters 216of the camera 201 to the image drawing unit 217 and the informationprocessing apparatus 400 (field of view information acquisition unit221).

FIG. 4 illustrates an example where the image input unit 210, the cameraposition and orientation management unit 211, the field of viewinformation management unit 214, and the image drawing unit 217 areincluded in the information processing apparatus 400. However, at leastsome of these units may be included in an apparatus other than theinformation processing apparatus 400 (for example, the HMD 200).

Next, differences in the details of the processing of the informationprocessing apparatus 400 according to the present exemplary embodimentfrom the first and second exemplary embodiments will be described withreference to the flowchart of FIG. 3.

(Step S304)

In step S304, the camera position and orientation management unit 211associates the image obtained in step S303 with the reference map 212 tocalculate a position and orientation of the camera 201 when the image iscaptured. The camera position and orientation management unit 211 alsoassociates the image obtained in step S303 with the reference mapswitching candidate 213 to calculate a position and orientation of thecamera 201 when the image is captured. The positions and orientations ofthe camera 201 are each calculated by the same method as described inthe first exemplary embodiment.

(Step S306)

In step S306, the field of view information acquisition unit 221obtains, from the field of view information management unit 214, fieldof view information for determining a change occurring in the appearanceof the virtual object as seen from the point of view of the camera 201when the reference map 212 is switched to the reference map switchingcandidate 213. The field of view information here includes the positionsand orientations of the camera 201 calculated in step S304 (thepositions and orientations of the camera 201 calculated by using thereference map 212 and the reference map switching candidate 213), thevirtual object information 215, and the internal parameters 216 of thecamera 201.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306. In the present exemplary embodiment,the reference map change determination unit 222 calculates a displayposition of the virtual object in the image captured by the camera 201,by using the position and orientation of the camera 201 calculated byusing the reference map 212 currently referred to. The reference mapchange determination unit 222 also calculates a display position of thevirtual object in the image captured by the camera 201, by using theposition and orientation of the camera 201 calculated by using thereference map switching candidate 213. If a difference between thedisplay positions of the virtual object is smaller than a predeterminedvalue, the reference map change determination unit 222 determines thatthe reference map 212 can be changed. If not, the reference map changedetermination unit 222 determines that the reference map 212 is not ableto be changed. The reference map change determination unit 222 storesthe determination result. The display positions of the virtual objectare calculated, for example, by multiplying the position and orientationof the virtual object by inverse matrices of the respective positionsand orientations of the camera 201 to calculate positions andorientations of the virtual object as seen from the respective points ofview of the camera 201, and multiplying the positions and orientationsof the virtual object by the internal parameter matrix of the camera201.

As described above, in the present exemplary embodiment, the referencemap 212 is determined to be able to be changed if a change occurring inthe display position of the virtual object when the reference map 212currently referred to is switched to the reference map switchingcandidate 213 is small. The reference map 212 can thus be switched bydetecting the timing when a sense of incongruity given to the user issmall without omission.

[Modification 3-1]

The present exemplary embodiment has described an example where thechange occurring in the appearance of the virtual object when thereference map 212 is switched to the reference map switching candidate213 is one occurring in the display position of the virtual object dueto the switching of the reference map 212. However, the change occurringin the appearance of the virtual object when the reference map 212 isswitched to the reference map switching candidate 213 is not limitedthereto. Whether the reference map 212 can be changed may be determinedbased on a change occurring in a moving direction of the displayposition of the virtual object due to the switching of the reference map212. The greater the difference between the moving direction of thedisplay position of the virtual object attributable to the motion of theuser wearing the HMD 200 and the moving direction of the displayposition of the virtual object attributable to the switching of thereference map 212 is, the more likely the following tendency is. Thatis, the user tends to be more likely to notice a change occurring in thedisplay position of the virtual object due to the switching of thereference map 212. The reference map 212 is therefore determined to beable to be changed if an angle formed between the moving directions ofthe display position of the virtual object due to the motion of the userwhen the reference map 212 is not switched and when the reference map212 is switched is smaller than a predetermined value. In such a manner,even if a change occurring in the display position of the virtual objectdue to the switching of the reference map 212 is large, the referencemap 212 can be switched by detecting the timing when a sense ofincongruity given to the user is small.

Differences of the processing of the information processing apparatus400 according to the present modification from that of the foregoingthird exemplary embodiment will be described with reference to theflowchart of FIG. 3.

(Step S307)

In step S307, the reference map change determination unit 222 determineswhether the reference map 212 can be changed, based on the field of viewinformation obtained in step S306. In the present modification, thereference map change determination unit 222 derives the moving directionof the display position of the virtual object in the image captured bythe camera 201 when the reference map 212 currently referred to is notswitched. The reference map change determination unit 222 also derivesthe moving direction of the display position of the virtual object inthe image captured by the camera 201 when the reference map 212 isswitched to the reference map switching candidate 213. If the angleformed between the two moving directions of the display position of thevirtual object is smaller than a predetermined value, the reference mapchange determination unit 222 determines that the reference map 212 canbe changed. If not, the reference map change determination unit 222determines that the reference map 212 is not able to be changed.

As described above, in the present modification, whether the referencemap 212 can be changed is determined based on a change occurring in themoving direction of the display position of the virtual object due tothe switching of the reference map 212. Even if the amount of movementof the display position of the virtual object resulting from theswitching of the reference map 212 is large, the reference map 212 canthus be switched by detecting the timing when a sense of incongruitygiven to the user is relatively small.

Next, a fourth exemplary embodiment will be described. The first tothird exemplary embodiments have described examples in which whether thereference map 212 can be changed is determined in consideration of theappearance of the virtual object as seen from the point of view of thecamera 201. However, whether the reference map 212 can be changed may bedetermined from only the position and orientation of the camera 201. Theuser is less likely to notice a change in the display position of thevirtual object if the user is changing the position and orientation ofthe camera 201 himself/herself. By switching the reference map 212 whena change in the position and orientation of the camera 201 is greaterthan a predetermine value, the reference map 212 can be switched intiming when a sense of incongruity given to the user is small.

The present exemplary embodiment thus differs from the first to thirdexemplary embodiments mainly in the method for determining whether thereference map 212 can be changed. In the description of the presentexemplary embodiment, parts similar to those of the first to thirdexemplary embodiments will be designated by the same reference numeralsas in FIGS. 1 to 4. A detailed description thereof will be omitted.

FIG. 5 is a diagram illustrating an example of a module configuration ofan information processing apparatus 500 according to the presentexemplary embodiment and an MR system in which the informationprocessing apparatus 500 is implemented. In the present exemplaryembodiment, the field of view information acquisition unit 221 accordingto the first to third exemplary embodiments is replaced with a cameraposition and orientation acquisition unit 521. The camera position andorientation acquisition unit 521 obtains the position and orientation ofthe camera 201 from the field of view information management unit 214.

FIG. 5 illustrates an example where the image input unit 210, the cameraposition and orientation management unit 211, the field of viewinformation management unit 214, and the image drawing unit 217 areincluded in the information processing apparatus 500. However, at leastsome of these units may be included in an apparatus other than theinformation processing apparatus 500 (for example, the HMD 200).

FIG. 6 is a flowchart illustrating an example of processing of the MRsystem. In the present exemplary embodiment, step S606 is performedinstead of step S303 according to the first to third exemplaryembodiments. In step S606, the camera position and orientationacquisition unit 521 obtains, from the field of view informationmanagement unit 214, the change speed of the position and orientation ofthe camera 201 at the time when the image obtained in step S303 iscaptured. The change speed is calculated by the camera position andorientation management unit 211 from a time series of changes in theposition and orientation of the camera 201, calculated in step S304. Aspecific example will be described. The camera position and orientationacquisition unit 521 obtains the magnitude of speed of a translationcomponent and the magnitude of speed of a rotation component (angularspeed) of the position and orientation of the camera 201.

In step S307, if the magnitude of the speed of the translation componentand the magnitude of the speed of the rotation component of the positionand orientation of the camera 201 are both greater than respectivepredetermined values, the reference map change determination unit 222determines that the reference map 212 can be changed. If not, thereference map change determination unit 222 determines that thereference map 212 is not able to be changed.

As described above, in the present exemplary embodiment, the referencemap 212 can be changed by detecting the timing when a sense ofincongruity given to the user is small from limited information of onlythe position and orientation of the camera 201.

[Modification 4-1]

The present exemplary embodiment has described an example in whichwhether the reference map 212 can be changed is determined based on thechange speed of the position and orientation of the camera 201 as atemporal change in the position and orientation of the camera 201.However, the temporal change in the position and orientation of thecamera 201 is not limited thereto. Whether the reference map 212 can bechanged may be determined based on the acceleration of change of theposition and orientation of the camera 201 as a temporal change in theposition and orientation of the camera 201. For example, in the momentof turning when the user wearing the HMD 200 makes a head-reciprocatingmotion, the user is less likely to be paying close attention to thevirtual object even if the speed of the rotation component of the camera201 is low. Therefore, if the magnitude of the acceleration of theposition and orientation of the camera 201 is greater than apredetermined value, the reference map 212 is determined to be able tobe changed. If not, the reference map 212 is determined to be unable tobe changed.

A specific example will be described. The camera position andorientation acquisition unit 521 calculates the magnitude of theacceleration of the translation component and the magnitude of theacceleration of the rotation component (angular acceleration) fromconsecutive positions and orientations of the camera 201. If themagnitudes are both greater than respective predetermined values, thereference map change determination unit 222 determines that thereference map 212 can be changed. If not, the reference map changedetermination unit 222 determines that the reference map 212 is not ableto be changed.

In such a manner, the reference map 212 can be switched in timing whenthe moving speed of the camera 201 is low and the user is unlikely tonotice a change in the display position of the virtual object, like whenthe user turns back in motion.

Next, a fifth exemplary embodiment will be described. The first tofourth exemplary embodiments have described examples in which whetherthe reference map 212 can be changed is determined based on the point ofview of the camera 201. However, if an attribute representing visibilityof the virtual object changes with time, whether the reference map 212can be changed may be determined from only the attribute representingthe visibility of the virtual object, without using the point of view ofthe camera 201. The present exemplary embodiment describes an example ofa method for determining whether the reference map 212 can be changed iftransparency of the virtual object in an arbitrary time is described inthe virtual object information 215 as a status representing thevisibility of the virtual object. If the transparency of the virtualobject is high, the user is less likely to notice a change in thedisplay position of the virtual object. By switching the reference map212 when the transparency of the virtual object is high, the referencemap 212 can be switched in timing when a sense of incongruity given tothe user is relatively small.

The present exemplary embodiment thus differs from the first to fourthexemplary embodiments mainly in the contents of the virtual objectinformation 215 and the method for determining whether the reference map212 can be changed. In the description of the present exemplaryembodiment, parts similar to those of the first to fourth exemplaryembodiments will be designated by the same reference numerals as inFIGS. 1 to 6. A detailed description thereof will be omitted.

FIG. 7 is a diagram illustrating an example of a module configuration ofan information processing apparatus 700 according to the presentexemplary embodiment and an MR system in which the informationprocessing apparatus 700 is implemented. In the present exemplaryembodiment, the field of view information acquisition unit 221 accordingto the first to third exemplary embodiments is replaced with a virtualobject attribute acquisition unit 721. The virtual object attributeacquisition unit 721 obtains the value of the attribute of the virtualobject included in the virtual object information 215 from the field ofview information management unit 214.

FIG. 7 illustrates an example where the image input unit 210, the cameraposition and orientation management unit 211, the field of viewinformation management unit 214, and the image drawing unit 217 areincluded in the information processing apparatus 700. However, at leastsome of these units may be included in an apparatus other than theinformation processing apparatus 700 (for example, the HMD 200).

FIG. 8 is a flowchart illustrating an example of processing of the MRsystem. In the present exemplary embodiment, step S806 is performedinstead of step S306 according to the first to third exemplaryembodiments. In step S806, the virtual object attribute acquisition unit721 obtains the transparency of the virtual object at processing timefrom the field of view information management unit 214 as the value ofthe attribute of the virtual object.

In step S307, if the value of the attribute (transparency) obtained instep S806 is greater than a predetermined value (if the transparency ishigher than the predetermined value), the reference map changedetermination unit 222 determines that the reference map 212 can bechanged. If not, the reference map change determination unit 222determines that the reference map 212 is not able to be changed.

As described above, in the present exemplary embodiment, the referencemap 212 can be switched by detecting the timing when a sense ofincongruity given to the user is small from limited information of onlythe attribute of the virtual object.

[Modification 5-1]

The present exemplary embodiment has described an example in whichwhether the reference map 212 can be changed is determined based on thetransparency, an optical attribute, of the virtual object as theattribute representing the visibility of the virtual object. However,the attribute representing the visibility of the virtual object is notlimited thereto. Whether the reference map 212 can be determined may bedetermined based on geometric information about the virtual object (forexample, the moving speed of the virtual object in a global coordinatesystem) as the attribute representing the visibility of the virtualobject. If the virtual object itself is in violent motion, the user isless likely to notice a change occurring in the display position of thevirtual object due to a change of the reference map 212 regardless ofthe motion of the user. If the magnitude of the moving speed of thevirtual object in the global coordinate system is greater than apredetermined value, the reference map 212 is then determined to be ableto be changed. If not, the reference map 212 is determined to be unableto be changed. In such a manner, the reference map 212 can be changed inappropriate timing when a sense of incongruity given to the user issmall.

Next, a sixth exemplary embodiment will be described. The first to fifthexemplary embodiments have described examples in which whether thereference map 212 can be changed is determined when the reference map212 is switched to a generated new map. However, the map to be switchedto does not necessarily need to be a generated new one. For example,suppose that a first map and a second map describing respectivedifferent spaces including an overlapping region are stored. Then, whenthe user moves from a region described by the first map to one describedby the second map in the environment where the MR system is used,whether the reference map 212 is able to be changed can be determined bya method similar to those described in the foregoing exemplaryembodiments in switching the reference map 212. Even in such an MRsystem of switching between a plurality of existing maps according tothe user's action, the reference map 212 can be switched in timing whena sense of incongruity given to the user is small. The first to fifthexemplary embodiments have also described examples of switching thereference map 212. However, in the foregoing exemplary embodiments, notthe timing to switch the reference map 212 but timing to performcorrection processing or modification processing for making a correctionor modification to a part of the reference map 212 may be determined.

Next, a seventh exemplary embodiment will be described. In the presentexemplary embodiment, any two or more of the determination results ofthe change determination methods of the reference map 212, described inthe first to fifth exemplary embodiments are combined to make a finaldetermination about whether the reference map 212 can be changed. Insuch a manner, the change determination method of the reference map 212can be adjusted according to the situation in which the MR system isused.

Specific examples include the following method. In the first exemplaryembodiment, the reference map 212 is changed if the virtual object isnot included in the field of view of the camera 201. In some situationswhere the virtual object is included in the field of view of the camera201, the reference map 212 can also be determined to be able to bechanged if the moving speed of the display position of the virtualobject is greater than a predetermined value as described in the secondexemplary embodiment. In such situations, the logical sum of the resultsof the respective determination methods described in the first andsecond exemplary embodiments is used as a final determination result.

Other specific examples include the following method. As described inmodification 1-1, the reference map 212 is changed if the displayposition of the virtual object is far from the center of the plane ofprojection of the camera 201. To further reduce the sense of incongruitygiven to the user, a high moving speed of the display position of thevirtual object may also be used as an additional condition as describedin the second exemplary embodiment. In such a situation, the logical ANDof the results of the respective determination methods described inmodification 1-1 and the second exemplary embodiment is used as a finaldetermination result. Similarly, the logical sum or logical AND of othermethods described in the first to fifth exemplary embodiments may beobtained to set various conditions. The present exemplary embodiment maybe applied to the sixth exemplary embodiment.

As described above, the determination results of a plurality ofdetermination methods may be combined to set a condition for determiningwhether the reference map 212 can be changed, according to the situationin which the MR system is used.

CONCLUSION

In the foregoing exemplary embodiments, the field of view informationobtained by the field of view information acquisition unit 221 may beinformation representing an appearance of the virtual object as seenfrom the point of view of the camera 201. An example is the combinationof the position and orientation of the camera 201, the position andorientation of the virtual object, the shape information about thevirtual object, and the internal parameters 216 of the camera 201described as the field of view information obtained in the firstexemplary embodiment. As described in modification 1-2, the field ofview information may further include the optical information about thevirtual object. In such a case, the optical information can also betaken into consideration to detect the timing to switch the referencemap 212 with fewer omissions.

In the foregoing exemplary embodiments, the reference map changedetermination unit 222 may use any method as long as the reference map212 is determined to be able to be changed when the user is unlikely tonotice a change that can occur in the appearance of the virtual objectdue to the change of the reference map 212, based on the field of viewinformation. An example is a method for determining the reference map212 to be able to be changed if the virtual object is not visible to thecamera 201 based on the static appearance of the virtual object from thepoint of view of the camera 201, obtained from the field of viewinformation.

Specifically, as described in the first exemplary embodiment, thereference map 212 may be determined to be able to be changed if thevirtual object is not included in the field of view of the camera 201.As described in modification 1-1, the reference map 212 may bedetermined to be able to be changed if the ratio of the display regionof the virtual object to the image captured by the camera 201 is small.Other examples include a method for determining the reference map 212 tobe able to be changed if the amount of change in the appearance of thevirtual object as seen from the point of view of the camera 201,obtained from the field of view information, is large. Specifically, asdescribed in the second exemplary embodiment, the reference map 212 maybe determined to be able to be changed if the amount of change per unittime of the display position of the virtual object in the image capturedby the camera 201 is large. As described in modification 2-1, thereference map 212 may be determined to be able to be changed if theamount of change per unit time of the orientation of the virtual objectas seen from the point of view of the camera 201 is large. Otherexamples include a method for calculating the amount of change per unittime occurring in the appearance of the virtual object in the imagecaptured by the camera 201 when a predetermined modification is actuallymade to the reference map 212, and determining the reference map 212 tobe able to be changed if the calculated amount of change is small.Specifically, as described in the third exemplary embodiment, thereference map 212 may be determined to be able to be changed if theamount of change per unit time occurring in the display position of thevirtual object in the image captured by the camera 201 due to the changeof the reference map 212 is small. As described in modification 3-1, thereference map 212 may be determined to be able to be changed if a changeoccurring in the moving direction of the display position of the virtualobject in the image captured by the camera 201 due to the change of thereference map 212 is small.

The foregoing are examples of variations of the determination methodswhen the field of view information includes only the arrangement andgeometric information about the camera 201 and the virtual object.However, the field of view information is not limited thereto. Forexample, if the field of view information includes the opticalinformation about the virtual object, the determination method describedin modification 1-2 may also be employed. That is, the reference map 212may be determined to be able to be changed if the virtual objectsuperimposed on the image captured by the camera 201 blends well intothe surrounding background.

The information used to determine whether the reference map 212 can bechanged may be an evaluation index for evaluating the user'sunlikeliness to notice a change occurring in the appearance of thevirtual object in the image captured by the camera 201 when thereference map 212 is changed to another map. An example is theinformation representing the appearance of the virtual object as seenfrom the point of view of the camera 201 as described in the first tothird exemplary embodiments. Other examples include a method fordetermining the reference map 212 to be able to be changed if a changein the position and orientation of the camera 201 per unit time islarge. Specifically, as described in the fourth exemplary embodiment,the reference map 212 may be determined to be able to be changed if themagnitude of the change speed of the position and orientation of thecamera 201 is large. As described in modification 4-1, the reference map212 may be determined to be able to be changed if the magnitude of theacceleration of change of the position and orientation of the camera 201is large. Other examples include the value of the attribute representingthe visibility of the virtual object. In such a case, the reference map212 is determined to be able to be changed if the visibility of thevirtual object is low.

The value of the attribute representing the visibility of the virtualobject may be a value by which the user's likeliness to notice a changeoccurring in the display position of the virtual object due to thechange of the reference map 212 can be evaluated. An example is thetransparency of the virtual object (if the transparency is high, thevisibility is low and the user is less likely to notice a change in thedisplay position of the virtual object) as described in the fifthexemplary embodiment. Other examples include the moving speed of thevirtual object itself (the higher the moving speed of the virtual objectis, the lower the visibility is and the less likely the user is tonotice a change in the display position of the virtual object) asdescribed in modification 5-1.

The foregoing exemplary embodiments are to be considered in all aspectsas illustrative of examples, and not restrictive of the technical scope,and may be implemented in various forms without departing from thetechnical concept or essential characteristics thereof.

Other Embodiments

Embodiment(s) can also be realized by a computer of a system orapparatus that reads out and executes computer executable instructions(e.g., one or more programs) recorded on a storage medium (which mayalso be referred to more fully as a ‘non-transitory computer-readablestorage medium’) to perform the functions of one or more of theabove-described embodiment(s) and/or that includes one or more circuits(e.g., application specific integrated circuit (ASIC)) for performingthe functions of one or more of the above-described embodiment(s), andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s) and/or controlling the one or morecircuits to perform the functions of one or more of the above-describedembodiment(s). The computer may comprise one or more processors (e.g.,central processing unit (CPU), micro processing unit (MPU)) and mayinclude a network of separate computers or separate processors to readout and execute the computer executable instructions. The computerexecutable instructions may be provided to the computer, for example,from a network or the storage medium. The storage medium may include,for example, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a storage of distributed computing systems, anoptical disk (such as a compact disc (CD), digital versatile disc (DVD),or Blu-ray Disc (BD)™), a flash memory device, a memory card, and thelike.

While exemplary embodiments have been described, it is to be understoodthat the scope of the following claims is to be accorded the broadestinterpretation so as to encompass all such modifications and equivalentstructures and functions.

This application claims the benefit of Japanese Patent Application No.2015-257146, filed Dec. 28, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus forperforming processing for superimposing a virtual object on an imagecaptured by an imaging apparatus based on a result of associationbetween a map and the image, the map representing an arrangement, in aspace in which the image is captured, of an indicator being associatedwith an indicator included in an image which is captured, and used forestimating a position and orientation of the imaging apparatus, theinformation processing apparatus comprising: one or more processors; andone or more memories storing instructions, when executed by the one ormore processors, causing the information processing apparatus tofunction as: an acquisition unit configured to obtain an evaluationindicator that is an indicator for evaluating the appearance of thevirtual object superimposed on the image captured by the imagingapparatus, wherein the evaluation indicator includes at least any one ofinformation representing the appearance of the virtual object as seenfrom a point of view of the imaging apparatus, information indicatingmotion of the imaging apparatus, and information indicating an attributerepresenting visibility of the virtual object; a determination unitconfigured to determine whether the map that is an existing map is ableto be changed to a newly generated map, which is generated by measuringthe space, based on the evaluation indicator for the existing map andestimated change of the evaluation indicator for the newly generatedmap; and a changing unit configured to change the map to the newlygenerated map in case where the determination unit determines that themap is able to be changed to the newly generated map.
 2. The informationprocessing apparatus according to claim 1, wherein the determinationunit is configured to make a determination about whether the virtualobject is included in an imaging range of the imaging apparatus, basedon the information representing the appearance of the virtual object asseen from the point of view of the imaging apparatus, and determinewhether the map is able to be changed, based on the determination. 3.The information processing apparatus according to claim 1, wherein thedetermination unit is configured to derive a ratio of a region of thevirtual object to a region of the image when the virtual object issuperimposed on the image, based on the information representing theappearance of the virtual object as seen from the point of view of theimaging apparatus, and determine whether the map is able to be changed,based on a result of comparison between the derived ratio and apredetermined value.
 4. The information processing apparatus accordingto claim 1, wherein the information representing the appearance of thevirtual object as seen from the point of view of the imaging apparatusincludes optical information about the virtual object, and wherein thedetermination unit is configured to determine whether the map is able tobe changed, based on a result of comparison between a value indicated bythe optical information about the virtual object and a predeterminedvalue.
 5. The information processing apparatus according to claim 1,wherein the information representing the appearance of the virtualobject as seen from the point of view of the imaging apparatus includesinformation representing a temporal change of the virtual object on aplane of projection of the image, and wherein the determination unit isconfigured to determine whether the map is able to be changed, based onthe information representing the temporal change of the virtual objecton the plane of projection of the image.
 6. The information processingapparatus according to claim 1, wherein information representing atemporal change of the virtual object on a plane of projection of theimage includes at least either one of a moving speed of a displayposition of the virtual object and a change speed of an orientation ofthe virtual object, and wherein the determination unit is configured todetermine whether the map is able to be changed, based on a result ofcomparison between at least either one of the moving speed of thedisplay position of the virtual object and the change speed of theorientation of the virtual object and a predetermined value.
 7. Theinformation processing apparatus according to claim 1, wherein theacquisition unit is configured to obtain the evaluation indicator thatis an indicator for evaluating the appearance of the virtual objectsuperimposed on the image by using the map that has not been changedyet, and the evaluation indicator that is an indicator for evaluating anappearance of the virtual object superimposed on the image by using acandidate map to be changed, and wherein the determination unit isconfigured to determine whether the map is able to be changed, based onthe evaluation indicator by using the map that has not been changed yetand estimated change of the evaluation indicator by using the candidatemap to be changed.
 8. The information processing apparatus according toclaim 7, wherein the evaluation indicator includes at least either oneof a display position of the virtual object and a moving direction ofthe display position of the virtual object.
 9. The informationprocessing apparatus according to claim 1, wherein the informationindicating the attribute representing the visibility of the virtualobject includes at least either one of optical information about thevirtual object and geometric information about the virtual object. 10.The information processing apparatus according to claim 1, wherein theinformation indicating the motion of the imaging apparatus includesinformation indicating a temporal change in a position and orientationof the imaging apparatus, and wherein the determination unit isconfigured to determine whether the map is able to be changed, based ona result of comparison between the temporal change in the position andorientation of the imaging apparatus and a predetermined value.
 11. Theinformation processing apparatus according to claim 10, wherein theinformation indicating the temporal change in the position andorientation of the imaging apparatus includes at least either one of achange speed and acceleration of the position and orientation of theimaging apparatus.
 12. The information processing apparatus according toclaim 1, wherein if the determination unit determines that the map isable to be changed, either the current map is switched to a new map or acorrection or modification is made to a part of the current map.
 13. Aninformation processing apparatus for performing processing forsuperimposing a virtual object on an image captured by an imagingapparatus based on a result of association between a map and the image,the map representing an arrangement, in a space in which the image iscaptured, of an indicator being associated with an indicator in an imagewhich is captured, and used for estimating a position and orientation ofthe imaging apparatus, the information processing apparatus comprising:one or more processors; and one or more memories storing instructions,when executed by the one or more processors, causing the informationprocessing apparatus to function as: an acquisition unit configured toobtain an evaluation indicator that is an indicator for evaluating theappearance of the virtual object superimposed on the image captured bythe imaging apparatus, wherein the evaluation indicator includes atleast any one of information representing the appearance of the virtualobject as seen from a point of view of the imaging apparatus,information indicating motion of the imaging apparatus, and informationindicating an attribute representing visibility of the virtual object; adetermination unit configured to determine whether the map that is anexisting map is able to be changed to a newly generated map, which isgenerated by measuring the space, based on a temporal change comparisonbetween the evaluation indicator of the existing map and the evaluationindicator of the newly generated map; and a changing unit configured tochange the map to the newly generated map in case where thedetermination unit determines that the map is able to be changed to thenewly generated map.
 14. An information processing method for performingprocessing for superimposing a virtual object on an image captured by animaging apparatus based on a result of association between a map and theimage, the map indicating an arrangement, in a space in which the imageis captured, of an indicator being associated with an indicator in animage which is captured, and used for estimating a position andorientation of the imaging apparatus, the information processing methodcomprising: obtaining an evaluation indicator that is an indicator forevaluating the appearance of the virtual object superimposed on theimage captured by the imaging apparatus, wherein the evaluationindicator includes at least any one of information representing theappearance of the virtual object as seen from a point of view of theimaging apparatus, information indicating motion of the imagingapparatus, and information indicating an attribute representingvisibility of the virtual object; determining whether the map that is anexisting map is able to be changed to a newly generated map, which isgenerated by measuring the space, based on the evaluation indicator forthe existing map and estimated change of the evaluation indicator forthe newly generated map; and changing the map to the newly generated mapin case where the determination unit determines that the map is able tobe changed to the newly generated map.
 15. An information processingmethod for performing processing for superimposing a virtual object onan image captured by an imaging apparatus based on a result ofassociation between a map and the image, the map representing anarrangement, in a space in which the image is captured, of an indicatorbeing associated with an indicator in an image which is captured, andused for estimating a position and orientation of the imaging apparatus,the information processing method comprising: obtaining an evaluationindicator that is an indicator for evaluating the appearance of thevirtual object superimposed on the image captured by the imagingapparatus, wherein the evaluation indicator includes at least any one ofinformation representing the appearance of the virtual object as seenfrom a point of view of the imaging apparatus, information indicatingmotion of the imaging apparatus, and information indicating an attributerepresenting visibility of the virtual object; determining whether themap that is an existing map is able to be changed to a newly generatedmap, which is generated by measuring the space, based on a temporalchange comparison between the evaluation indicator of the existing mapand the evaluation indicator of the newly generated map; and changingthe map to the newly generated map in case where the determination unitdetermines that the map is able to be changed to the newly generatedmap.
 16. A non-transitory computer-readable medium storing a program forcausing a computer to function as units of an information processingapparatus for performing processing for superimposing a virtual objecton an image captured by an imaging apparatus based on a result ofassociation between a map and the image, the map representing anarrangement, in a space in which the image is captured, of an indicatorbeing associated with an indicator in an image which is captured, andused for estimating a position and orientation of the imaging apparatus,the information processing apparatus including: an acquisition unitconfigured to obtain an evaluation indicator that is an indicator forevaluating the appearance of the virtual object superimposed on theimage captured by the imaging apparatus, wherein the evaluationindicator includes at least any one of information representing theappearance of the virtual object as seen from a point of view of theimaging apparatus, information indicating motion of the imagingapparatus, and information indicating an attribute representingvisibility of the virtual object; a determination unit configured todetermine whether the map that is an existing map is able to be changedto a newly generated map, which is generated by measuring the space,based on the evaluation indicator for the existing map and estimatedchange of the evaluation indicator for the newly generated map; and achanging unit configured to change the map to the newly generated map incase where the determination unit determines that the map is able to bechanged to the newly generated map.
 17. A storage non-transitorycomputer-readable medium storing a program for causing a computer tofunction as units of an information processing apparatus for performingprocessing for superimposing a virtual object on an image captured by animaging apparatus based on a result of association between a map and theimage, the map representing an arrangement, in a space in which theimage is captured, of an indicator being associated with an indicator inan image which is captured, and used for estimating a position andorientation of the imaging apparatus, the information processingapparatus including: an acquisition unit configured to obtain anevaluation indicator that is an indicator for evaluating the appearanceof the virtual object superimposed on the image captured by the imagingapparatus, wherein the evaluation indicator includes at least any one ofinformation representing the appearance of the virtual object as seenfrom a point of view of the imaging apparatus, information indicatingmotion of the imaging apparatus, and information indicating an attributerepresenting visibility of the virtual object; a determination unitconfigured to determine whether the map that is an existing map is ableto be changed to a newly generated map, which is generated by measuringthe space, based on a temporal change comparison between the evaluationindicator of the existing map and the evaluation indicator of the newlygenerated map; and a changing unit configured to change the map to thenewly generated map in case where the determination unit determines thatthe map is able to be changed to the newly generated map.